…line

The current Kafka message key projects fullDocument.value.key, which is
null on update events since BB-355 removed
change.stream.full.document=updateLookup. Every update for a given bucket
therefore serialises to the same key Struct
({ns:{coll:<bucket>}, fullDocument:null}) and lands on one partition,
breaking per-object ordering across op types (insert vs update) and
pinning a hot bucket's update traffic to a single partition (a blocker
for the oplog scaling work tracked in BB-756).

Fix: insert a $addFields stage into the MongoDB connector's
change-stream pipeline (right after $match, before the conditional
location-strip $set) that synthesises a top-level 'key' field by
$ifNull coalescing $fullDocument.value.key and
$updateDescription.updatedFields.value.key. The connector's
output.schema.key replaces the broken fullDocument nested record with a
sibling 'key' field; the existing 'ns:{coll}' projection is preserved,
so the Kafka message key remains {ns:{coll:<bucket>}, key:<object-key>}
— bucket-level isolation is unchanged. All events for the same logical
S3 object (insert, master/version updates, replication-status updates,
delete-marker updates) yield the same key value and hash to the same
partition. Master and version documents share the same value.key, so
master/version events also collapse to the same partition without
prefix-stripping.

The $ifNull variant relies on metadata always writing the whole 'value'
subdocument (full $set). Confirmed against arsenal: there is no partial
dotted-$set path for object MD today. A hypothetical future partial
$set would not populate updateDescription.updatedFields.value.key and
that event would mis-partition — accepted risk per the ticket
discussion.

The change is propagated to existing connectors via the existing
in-place PUT /connectors/{name}/config reconciliation (no recreate, no
resume-token loss — the change touches the key schema + the added
$addFields stage only, not the $match stage). Downstream oplog consumers
do not read the Kafka message key, so the new key shape is
consumer-transparent.

History / discussion:

  https://scality.atlassian.net/browse/BB-768?focusedCommentId=477122

This supersedes the earlier SMT-based approach (a Kafka Connect Single
Message Transform deriving the key from documentKey._id with
master/version prefix-stripping), which would have spanned three repos
and added a new Java artifact + operator feature flag. Closed in favour
of this single-repo fix after a per-event-cost measurement showed the
$addFields adds ~600–900 ns/event ≈ ~1–2% of one core at 20k ops/s on
the mongod — not a throughput concern at our target rates.

Superseded work (to be closed):

  #2741   (SMT-track Backbeat PR)
  scality/Zenko#2410      (ZENKO-5274 — Java SMT in kafka-connect image)
  https://scality.atlassian.net/browse/ZKOP-553   (operator feature flag — no longer needed)

Issue: BB-768

…sed key

Adds a Prometheus counter `s3_oplog_event_missing_key_total` (labelled by
operationType) that increments when a consumed oplog event reaches
downstream processing in ListRecordStream but lacks the top-level 'key'
field synthesised by the connector pipeline in the previous commit.

Should stay at zero in steady-state. A non-zero rate signals a regression
— e.g. a future write path on object MD that doesn't $set the whole
'value' subdocument and so bypasses the $ifNull coalesce.

Wired as a new `KafkaLogConsumerMetrics` module mirroring the
LifecycleMetrics pattern (static class, per-method try/catch +
handleError so a prometheus-side failure can't propagate into the
oplog read path).

Per-PR review feedback.

Issue: BB-768